Object detection apparatus, object detection method, and computer-readable recording medium

ABSTRACT

An object detection apparatus 100 is provided with: a fish-eye image acquisition unit 10 configured to acquire a time series fish-eye image; a horizontal panorama image generation unit 20 configured to, for each frame, perform conversion to a horizontal panorama image in which a vertical direction in a real space is expressed in a perpendicular direction of the frame, and an azimuth is expressed equiangularly in a horizontal direction of the frame; an edge pair extraction unit 30 configured to extract a pair of edges in the perpendicular direction from the horizontal panorama image; a change rate extraction unit 40 configured to extract a change rate of an inter-edge distance between the pair of edges; a lower end region extraction unit 50 configured to extract a region of a lower end of an object providing the pair of edges; a distance change rate extraction unit 60 configured to calculate a distance from the object to the fish-eye camera based on the position of the region of the lower end of the object in the horizontal panorama image, and extract a change rate of the distance; and an object detection unit 70 configured to determine whether or not the object exists based on the change rate of the inter-edge distance and the change rate of the distance.

TECHNICAL FIELD

The present invention relates to an object detection apparatus and anobject detection method for detecting an object from an image obtainedby a fish-eye camera, and further relates to a computer-readablerecording medium where a program for realizing these is recorded.

BACKGROUND ART

In recent years, there have been disclosed several apparatusesconfigured to detect an object that exists in the vicinity of a vehiclebased on an image from a camera installed in the vehicle (hereinafter,referred to as an “in-vehicle camera”) to assist a driver when thevehicle is parked (for example, see Patent Documents 1 to 3).

Specifically, Patent Document 1 discloses an apparatus configured todetect a road attachment, for example a road sign, based on a capturedimage from an in-vehicle camera. The apparatus disclosed in PatentDocument 1 detects a column-shaped columnar region by counting thefrequency of edge pixels arranged in a vertical direction in thecaptured image, and further detects a region with a specific color and aregion with a specific shape in the image, and detects a target roadsign based on each of the detected regions.

Patent Document 2 discloses an apparatus configured to detect anobstacle based on a captured image from an in-vehicle camera arranged soas to capture an image forward or rearward of a vehicle. The apparatusdisclosed in Patent Document 2 extracts an edge image from a capturedimage, averages the extracted edge image with a previously extractededge image, and, based on the averaged edge image, detects an obstaclelocated forward or rearward of the vehicle.

Patent Document 3 also discloses an apparatus configured to detect anobject in the vicinity of a vehicle based on a captured image from anin-vehicle camera. However, in the apparatus disclosed in PatentDocument 3, image conversion is performed on the captured image from thein-vehicle camera to generate an image captured from another cameraviewpoint. Ordinarily, the mounting position of the in-vehicle cameradiffers depending on the vehicle type, and as a result the appearance ofthe object differs. Therefore, fundamentally, it is necessary to createan identification device for object detection for each vehicle type,which increases cost, but according to the apparatus disclosed in PatentDocument 3, this increase in cost can be suppressed.

Therefore, by installing an apparatus according to Patent Documents 1 to3 in a vehicle, when parking, an object in the vicinity of the vehicleis detected, so the driver can park with peace of mind, and also safetyin the vicinity of the vehicle is secured.

LIST OF RELATED ART DOCUMENTS Patent Documents

Patent Document 1: Japanese Patent Laid-Open Publication No. 2016-162436

Patent Document 2: Japanese Patent Laid-Open Publication No. 2005-44196

Patent Document 3: Japanese Patent Laid-Open Publication No. 2011-210087

SUMMARY OF INVENTION Problems to be Solved by the Invention

However, with the apparatuses disclosed in above Patent Documents 1 to3, there is the problem that it is difficult to detect a rod-shapedobject such as a pole, which is likely to become an obstacle whenparking, in a case where the apparatus is combined with a fish-eyecamera. Installation of fish-eye cameras in vehicles in order to provideparking assistance has been advancing. This will be specificallydescribed below.

First, its object detection using a camera installed at the front of avehicle, as disclosed in Patent Documents 1 and 2 described above, it iscommon to detect an object using a straight edge in a captured image asa clue. Also, in the in-vehicle camera in this case, the lens has acomparatively narrow viewing angle, so there is little lens distortion,and the mounting angle is an angle close to horizontal. Therefore, it iseasy to extract a straight vertical edge and a horizontal edge.

On the other hand, in a fish-eye camera, the viewing angle exceeds 180degrees, and an image from the fish-eye camera includes a large amountof barrel-shaped distortion, such that a line segment that isessentially straight in a real space is observed to curve in the image.Therefore, in the apparatus, it is difficult to extract a vertical edgeor a horizontal edge from a captured image.

In order to facilitate extraction of a vertical edge and a horizontaledge, it is conceivable to adopt processing to correct lens distortionas pre-processing, but an in-vehicle camera mounted in order to provideparking assistance often is installed such that an optical axisdirection is oriented obliquely downward. Therefore, even if aconventional distortion correction method is adopted, as shown in FIGS.11 and 12, only an image with a large amount of projective distortioncan be obtained, so it is still difficult to extract a vertical edge anda horizontal edge. FIG. 11 shows an example of a rod-shaped object in areal space. In FIG. 11, reference numeral 201 denotes a rod-shapedobject. FIG. 12 shows a state in which conventional distortioncorrection has been performed on a rod-shaped object whose image wascaptured with a fish-eye lens. In FIG. 12, reference numeral 301 denotesa pair of line segments indicating the outer shape of the rod-shapedobject 201.

On the other hand, it is also possible to correct the distortion byvirtually orienting the optical axis direction of the camera, which isoriented obliquely downward, in the horizontal direction. When suchcorrection is performed, as shown in FIG. 13, the image of therod-shaped object 201 is converted into a straight shape in theperpendicular direction. FIG. 13 shows an example of a captured imageobtained by virtually setting the optical axis direction of the camerato the horizontal direction. However, in the correction shown in FIG.13, even if the size of the image in the horizontal direction afterimage conversion is infinite, a field of view exceeding a viewing angleof 180 degrees cannot be expressed in one image, so this sort ofcorrection is not suitable for an object detection apparatus installedin a vehicle.

Also, in the technology disclosed in Patent Document 3, an image thatappears similar to an image when viewed from another camera viewpoint isgenerated by assuming the shape of an object and the position where theobject exists in real space. Therefore, in order to perform imageconversion using the technology disclosed in Patent Document 3, it isnecessary to assume various existence positions of the object, and thereis a problem that this requires trial and error.

An example object of the invention is to provide an object detectionapparatus an object detection method, and a computer-readable recordingmedium that address the above-described problems, such that it ispossible to easily detect a rod-shaped object from a captured image evenin a case where a fish-eye camera was used.

Means for Solving the Problems

In order to achieve the example object described above, an objectdetection apparatus according to an example aspect of the inventionincludes:

a fish-eye image acquisition unit configured to acquire a time seriesfish-eye image output from a fish-eye camera;

a horizontal panorama image generation unit configured to, for eachframe included in the time series fish-eye image, perform conversion toa horizontal panorama image in which a vertical direction in a realspace is expressed in a perpendicular direction of the frame, and anazimuth is expressed equiangularly in a horizontal direction of theframe;

an edge pair extraction unit configured to extract, for each frame, apair of edges in the perpendicular direction from the horizontalpanorama image;

a change rate extraction unit configured to extract, between the frames,a change rate of an inter-edge distance between the pair of edgesextracted;

a lower end region extraction unit configured to extract, for eachframe, a region that corresponds to a lower end of an object predictedto be providing the pair of edges;

a distance change rate extraction unit configured to calculate, for eachframe, a distance from the object to the fish-eye camera based on theposition of the region that corresponds to the lower end of the objectin the horizontal panorama image, and extract, between the frames, achange rate of the distance from the object to the fish-eye camera; and

an object detection unit configured to determine whether or not theobject exists based on the change rate of the inter-edge distance andthe change rate of the distance from the object to the fish-eye camerathat were extracted.

Also, in order to achieve the example object described above, an objectdetection method according to an example aspect of the inventionincludes:

(a) a step of acquiring a time series fish-eye image output from afish-eye camera;

(b) a step of, for each frame included in the time series fish-eyeimage, performing conversion to a horizontal panorama image in which avertical direction in a real space is expressed in a perpendiculardirection of the frame, and an azimuth is expressed equiangularly it ahorizontal direction of the frame;

(c) a step of extracting, for each frame, a pair of edges in theperpendicular direction from the horizontal panorama image;

(d) a step of extracting, between the frames, a change rate of aninter-edge distance between the pair of edges extracted;

(e) a step of extracting, for each frame, a region that corresponds to alower end of an object predicted to be providing the pair of edges;

(f) a step of calculating, for each frame, a distance from the object tothe fish-eye camera based on the position of the region that correspondsto the lower end of the object in the horizontal panorama image, andextracting, between the frames, a change rate of the distance from theobject to the fish-eye camera; and

(g) a step of determining whether or not the object exists based on thechange rate of the inter-edge distance and the change rate of thedistance from the object to the fish-eye camera that were extracted.

Furthermore, in order to achieve the example object described above, acomputer-readable recording medium according to an example aspect of theinvention includes a program recorded thereon, the program includinginstructions that cause a computer to carry out:

(a) a step of acquiring a time series fish-eye image output from afish-eye camera:

(b) a step of, for each frame included in the time series fish-eyeimage, performing conversion to a horizontal panorama image in which avertical direction in a real space is expressed in a perpendiculardirection of the frame, and an azimuth is expressed equiangularly in ahorizontal direction of the frame;

(c) a step of extracting, for each frame, a pair of edges in theperpendicular direction from the horizontal panorama image;

(d) a step of extracting, between the frames, a change rate of aninter-edge distance between the pair of edges extracted;

(e) a step of extracting, for each frame, a region that corresponds to alower end of an object predicted to be providing the pair of edges;

(f) a step of calculating, for each frame, a distance from the object tothe fish-eye camera based on the position of the region that correspondsto the lower end of the object in the horizontal panorama image, andextracting, between the frames, a change rate of the distance from theobject to the fish-eye camera; and

(g) a step of determining whether or not the object exists based on thechange rate of the inter-edge distance and the change rate of thedistance from the object to the fish-eye camera that were extracted.

Advantageous Effects of the Invention

As described above, according to the invention, it is possible to easilydetect a rod-shaped object from a captured image even in a case where afish-eye camera was used.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a schematic configuration of an objectdetection apparatus according to an example embodiment of the invention.

FIG. 2 is a block diagram more specifically showing the configuration ofthe object detection apparatus according to an example embodiment of theinvention.

FIG. 3 schematically shows an example of a subject fish-eye image in anexample embodiment of the invention.

FIG. 4 shows an example of a parallelized viewpoint coordinate systemdefined in an example embodiment of the invention.

FIG. 5 shows two perspective projection corrected images with differentparallelization viewpoints, and a horizontal panorama image obtainedfrom the perspective projection corrected images.

FIG. 6 shows an example of a fish-eye image to be processed and anexample of a horizontal panorama image generated from that fish-eyeimage, in an example embodiment of the invention.

FIG. 7 shows a conceptual image obtained when an image of a real spacecaptured using a fish-eye camera is viewed from above.

FIG. 8 shows a horizontal panorama image of the real space shown in FIG.7.

FIG. 9 is a flowchart showing operation of the object detectionapparatus according to an example embodiment of the invention.

FIG. 10 is a block diagram showing an example of a computer thatrealizes the object detection apparatus according to an exampleembodiment of the invention.

FIG. 11 shows an example of a rod-shaped object in a real space.

FIG. 12 shows a state in which conventional distortion correction hasbeen performed on a rod-shaped object whose image was captured with afish-eye lens.

FIG. 13 shows an example of an image captured using a fish-eye lens,obtained by virtually setting an optical axis direction of a camera tothe horizontal direction.

EXAMPLE EMBODIMENT Example Embodiment

Following is a description of an object detection apparatus, an objectdetection method, and a computer-readable recording medium according toan example embodiment of the invention, with reference to FIGS. 1 to 10.

[Apparatus Configuration]

First, the configuration of an object detection apparatus according tothis example embodiment will be described. FIG. 1 is a block diagramshowing a schematic configuration of an object detection apparatusaccording to an example embodiment of the invention.

As shown in FIG. 1, an object detection apparatus 100 according to thisexample embodiment is an apparatus for detecting an object from a timeseries fish-eye image output from a fish-eye camera. As shown in FIG. 1,the object detection apparatus 100 includes a fish-eye image acquisitionunit 10, a horizontal panorama image generation unit 20, an edge pairextraction unit 30, a change rate extraction unit 40, a lower end regionextraction unit 50, a distance change rate extraction unit 60, and anobject detection unit 70.

The fish-eye image acquisition unit 10 is configured to acquire a timeseries fish-eye image output from a fish-eye camera. The horizontalpanorama image generation unit 20 is configured to, for each frameincluded in the time series fish-eye image, perform conversion to ahorizontal panorama image in which a vertical direction in a real spaceis expressed in a perpendicular direction of the frame, and an azimuthis expressed equiangularly in a horizontal direction of the frame.

The edge pair extraction unit 30 is configured to extract, for eachframe, a pair of edges in the perpendicular direction from thehorizontal panorama image. The change rate extraction unit 40 isconfigured to extract, between the frames, a change rate of aninter-edge distance between the pair of edges extracted (hereinafterreferred to as the “pair of edges”).

The lower end region extraction unit 50 is configured to extract, foreach frame, a region that corresponds to a lower end of an objectpredicted to be providing the pair of edges. The distance change rateextraction unit 60 is configured to calculate, for each frame, adistance from the object to the fish-eye camera based on the position ofthe region that corresponds to the lower end of the object in thehorizontal panorama image. Also, the distance change rate extractionunit 60 is configured to extract, between the frames, a change rate ofthe distance from the object to the fish-eye camera.

The object detection unit 70 is configured to determine whether or notthe object exists based on the change rate of the inter-edge distancethat was extracted by the change rate extraction unit 40, and the changerate of the distance from the object to the fish-eye camera that wasextracted by the distance change rate extraction unit 60.

Thus, in this example embodiment, the horizontal panorama image in whichthe vertical direction in the real space is expressed in theperpendicular direction of the frame is generated from the fish-eyeimage output from the fish-eye camera. Therefore, the pair of edges inthe perpendicular direction can easily be extracted. Also, in thehorizontal panorama image, the azimuth is expressed equiangularly in thehorizontal direction of the frame, so a visual field with a viewingangle of 180 degrees is also expressed by one image. Therefore,according to this example embodiment, it is possible to easily detect arod-shaped object from a captured image even in a case where thefish-eye camera was used.

Next, the object detection device 100 according to this exampleembodiment will be described more specifically with reference to FIG. 2.FIG. 2 is a block diagram more specifically showing the configuration ofthe object detection apparatus according to an example embodiment of theinvention.

As shown in FIG. 2, in this example embodiment, the object detectiondevice 100 is connected to a fish-eye camera 80. The fish-eye camera 80is an imaging device provided with a fish-eye lens system as an opticalsystem. The fish-eye camera 80 outputs a captured image (a fish-eyeimage) at a frame rate that has been set. Also, the fish-eye camera 80,in a vehicle, enables detection of a rod-shaped object that exists itsthe vicinity of the vehicle, so the fish-eye camera 80 is disposed suchthat, in the vehicle, an optical axis direction is a direction inclineddownward (a direction obliquely downward) in a vertical direction from ahorizontal plane. Note that the fish-eye camera 80 may be disposed atany position in the vehicle.

The horizontal panorama image generation unit 20, in this exampleembodiment, generates an image (a horizontal panorama image) obtained byrotating a camera whose optical axis is oriented in the horizontaldirection about 180 degrees in a horizontal plane from the imageobtained by the fish-eye camera 80, which has a depression angle.Therefore, the horizontal panorama image generation unit 20 is providedwith a viewpoint compensation vector generation unit 21 and an imagegeneration unit 22.

The viewpoint compensation vector generation unit 21, based on a rollangle around the optical axis of the fish-eye camera 80 and a pitchangle of the optical axis with respect to a plane parallel to a groundplane of the object to be detected, generates a viewpoint compensationvector that converts the frame to an image obtained by capturing animage of the object from a direction parallel to the ground plane. Notethat the roll angle and the pitch angle of the fish-eye camera 80 may beset values, or may be values acquired from a sensor installed in thevehicle.

Specifically, the viewpoint compensation vector generation unit 21generates, as the viewpoint compensation vector, a relative vectorbetween an optical axis vector of the fish-eye camera 80 and a vectorparallel to the ground plane of the object. This relative vector is avector that expresses rotation between two coordinate systems. Examplesof a rotation expression method commonly include Quaternion expression,Euler angle expression, and the like. In this example embodiment, anyexpression method may be adopted.

First, the image generation unit 22, with respect to the frame that wasconverted using the viewpoint compensation vector, sets a plurality ofviewpoints parallel to the ground plane in the horizontal direction ofthe frame. Also, for each of the viewpoints that was set, on the frameafter conversion, based on a coordinate system that includes a sightline from each viewpoint as an axis, the image generation unit 22performs distortion correction by perspective projection approximation.In addition, using an image element in the perpendicular directionextracted from each of the frames after correction, the image generationunit 22 generates one new frame, and adopts this as the horizontalpanorama image.

Specifically, the image generation unit 22 first sets a set ofparallelized viewpoints using a viewpoint compensation vector V withrespect to the frame of the fish-eye image. Next, the image generationunit 22 arbitrarily divides the visual field range in the horizontaldirection, and executes distortion correction by perspective projectionapproximation in each parallelized viewpoint coordinate system of eachset of parallelized viewpoints (each parallelized viewpoint row). Then,the image generation unit 22 arranges image elements in a perpendiculardirection passing through the center of each viewpoint in the horizontaldirection in the order of the parallelized viewpoint rows, and byconnecting these, generates a single composite image.

Here, the processing of the image generation unit 22 will be describedin more detail with reference to FIGS. 3 to 5. FIG. 3 schematicallyshows an example of a subject fish-eye image in an example embodiment ofthe invention. FIG. 4 shows an example of a parallelized viewpointcoordinate system defined in an example embodiment of the invention.FIG. 5 shows two perspective projection corrected images with differentparallelization viewpoints, and a horizontal panorama image obtainedfrom these perspective projection corrected images.

In the example of FIG. 3, a fish-eye image I_(F) shows people (Person A,Person B, and Person C) image-captured from a downward-directedviewpoint with the ground used as the ground plane. As shown in FIG. 4,the image generation unit 22 sets a set of parallelized viewpoints φ_(n)using the viewpoint compensation vector V, and further sets aparallelized viewpoint coordinate system for each parallelized viewpointφ_(n), and executes distortion correction by perspective projectionapproximation. Thus, a perspective projection corrected image I_(P)^((φn)) is set for each parallelized viewpoint φ_(n).

Note that n is a natural number indicating the number of parallelizedviewpoints, and i and j in FIG. 4 indicate natural numbers of n or less.In FIG. 4, O^((φn)) indicates a limit of the viewpoint compensationvector, and z^((φn)) indicates the sight line that passes through eachparallelized viewpoint. Also, u^((φn)) and v^((φn)) indicate respectiveaxes the parallelized viewpoint coordinate system.

As shown in FIG. 5, the image generation unit 22 slices a centralportion of the perspective projection corrected image I_(P) ^((φn)) inthe vertical direction, and cuts out a slice image I_(S) ^((φn)). Theslice image I_(S) ^((φn)) becomes a vertical image element that passesthrough the center of each viewpoint. Afterward, the image generationunit 22 combines the slice images I_(S) ^((φn)) into one frame. Thus, afinal horizontal panorama image is generated.

The horizontal panorama image obtained in this way has thecharacteristic properties described in (1) to (3) below. Hereinafter,the characteristic properties (1) to (3) of the horizontal panoramaimage will be described with reference to FIGS. 6 to 8. FIG. 6 shows anexample of a fish-eye image to be processed and an example of ahorizontal panorama image generated from that fish-eye image, in anexample embodiment of the invention. FIG. 7 shows a conceptual imageobtained when an image of a real space captured using a fish-eye camerais viewed from above. FIG. 8 shows a horizontal panorama image of thereal space shown in FIG. 7.

(1) As shown in FIG. 6, in the fish-eye image, an outer edge of arod-shaped object 201 is observed as a pair of curved line segments 301.On the other hand, because the horizontal panorama image is generated bycompositing the slice images I_(S) ^((φn)), the outer edge of therod-shaped object 201 is observed as a pair of straight line segments401 in the horizontal panorama image. In the horizontal panorama image,a line segment oriented in the vertical direction in the real space isconverted into a line segment oriented in the perpendicular direction.

(2) In the horizontal panorama image, one image covers a wide field ofview of a fish-eye image exceeding 180 degrees, and scale distortion inthe horizontal direction is eliminated. Furthermore, an x-coordinate inthe horizontal panorama image corresponds one-to-one with the cameraazimuth when the camera is virtually rotated on a horizontal plane.

Specifically, as shown in FIG. 7, in a real space, a virtual circle 602centered on an optical center 601 of the fish-eye camera 80 is assumed,and points A to G are assumed to exist at every 30 degrees from aposition 603 at 9 o'clock to a position 604 at 3 o'clock on this virtualcircumference. When a horizontal panorama image is generated from thefish-eye image in the real space shown in FIG. 7, the result is as shownin FIG. 8. That is, as shown in FIG. 8, the x-coordinates of the pointsA to G correspond one-to-one with the azimuth of the points A to G shownin FIG. 7, and each azimuth is equally allocated on the x-axis.

(3) The distance from the fish-eye camera 80 to the object in the realspace (on a road plane coordinate system) corresponds one-to-one withthe y-coordinate in the horizontal panorama image.

Specifically, as shown in FIG. 7, a virtual circle 605 and a virtualcircle 606 centered on the optical center 601 are assumed. At this time,if the radii of the circle 602, the circle 605, and the circle 606 arer1, r2, and r3, respectively, a relationship of r1>r2>r3 is established.In this case, a point on the circumference of the circle 602, as shownin FIG. 8, in the horizontal panorama image, is located on a horizontalline 701. Similarly, a point on the circumference of the circle 605 islocated on a horizontal line 702, and a point on the circumference ofthe circle 606 is located on a horizontal line 703. That is, thedistance from the fish-eye camera to the object corresponds one-to-onewith the y-coordinate in the horizontal panorama image.

Note that the relationship between the distance from the optical center601 in the real space and the y-coordinate on the horizontal panoramaimage is uniquely determined depending on the internal parameters andexternal parameters of the fish-eye camera, the size of the horizontalpanorama image, the resolution of the horizontal panorama image, and thelike.

The processing by each of the edge pair extraction unit 30, the changerate extraction unit 40, the lower end region extraction unit 50, thedistance change rate extraction unit 60, and the object detection unit70 is performed by utilizing the characteristic properties of (1) to (3)above.

In this example embodiment, the edge pair extraction unit 30 appliesedge extraction processing to the horizontal panorama image to extractan edge oriented in the perpendicular direction of the image.Furthermore, the edge pair extraction unit 30 also extracts another edgeoriented in the perpendicular direction positioned near the extractededge, sets both edges as a pair of edges, and outputs a rectangularregion where the pair of edges exist.

An example of the edge pair extraction method is a method in which anedge pixel is obtained by applying a local edge extraction filter in anorthodox manner, and then a straight line extraction algorithm isapplied. Another example is a method of directly obtaining the pixelposition of a pair of edges using a neural network or the like. The edgepair extraction unit 30 can also extract a plurality of sets of edgepairs. In that case, the subsequent processing may be executedindependently for each pair.

In this example embodiment, the change rate extraction unit 40 specifiesthe distance between edges of the edge pair extracted by the edge pairextraction unit 30 on a pixel-by-pixel basis in the image for eachframe. Also, for example, a difference between the inter-edge distancesin two frames is calculated, and a change rate of the inter-edgedistance is extracted based on the calculated difference. Note that thedifference between the inter-edge distances may also be calculated usingthree or more frames.

In this example embodiment, the lower end region extraction unit 50searches the vicinity of a portion on the lower side of a screen of therectangular boundary region output by the edge pair extraction unit 30,extracts a region that corresponds to the lower end of the rod-shapedobject, and specifies the position of the extracted region.

Specifically, the lower end of the rod-shaped object usually has aspecific shape such as a downwardly convex arc shape in the horizontalpanorama image. Therefore, the lower end region extraction unit 50 firstexecutes edge feature extraction in a peripheral region of the portionon the lower side of the screen of the rectangular region. Next, thelower end region extraction unit 50 executes sliding window processingwith respect to the edge extracted by the edge feature extraction usinga discriminator that has learned the shape of the lower end of therod-shaped object in advance, and specifies an edge that corresponds tothe lower end of the rod-shaped object, and specifies coordinates of thespecified edge in the horizontal panorama image. Note that in a casewhere a value of certainty or similarity output from the discriminatordoes not exceed a predetermined threshold, the lower end regionextraction unit 50 determines that the rod-shaped object does not exist.

In this example embodiment, the distance change rate extraction unit 60first calculates the distance from the rod-shaped object to the fish-eyecamera 80 based on the coordinates of the region that corresponds to thelower end of the rod-shaped object extracted by the lower end regionextraction unit 50 in the horizontal panorama image.

Specifically, as shown in FIG. 7, the distance from the fish-eye camera80 to the object corresponds to the y-coordinate in the horizontalpanorama image on a one-to-one basis, and the relationship between thisdistance and the y-coordinate is uniquely determined. Therefore, thedistance change rate extraction unit 60 uses the relationship betweenthis distance and the y-coordinate to convert the coordinates of theregion that corresponds to the lower end of the rod-shaped object in thehorizontal panorama image to the distance from the fish-eye camera 80 tothe object.

Also, the relationship between the distance from the fish-eye camera 80to the object and the y-coordinate may be provided by, for example, alook-up table created by combining specific numerical values of thedistance and the y-coordinate.

Next, the distance change rate extraction unit 60 calculates, withrespect to frames extracted by the change rate extraction unit 40,between frames, the difference in distances from the fish-eye camera 80to the rod-shaped object, and based on the calculated difference,extracts a change rate in the distance from the fish-eye camera 80 tothe rod-shaped object. Note that, as described above, with respect toany of the frames to be extracted, when the lower end region extractionunit 50 has determined that the rod-shaped object does not exist, thedistance change rate extraction unit 60 determines that a change rate ofthe distance cannot be extracted.

As described above, the object detection unit 70 determines whether ornot the object exists based on the change rate of the distance betweenedge pairs and the change rate of the distance from the object to thefish-eye camera 80. This will be specifically described below.

First, the interval between the edge pairs is proportional to theazimuth difference between the side surfaces of the object thatcorresponds to the two edges, in other words, the angle. Also, there isa proportional relationship of l=r×θ between the radius r of the arc,the central angle θ, and the arc length l. Therefore, the arc length lis approximated to the diameter of the rod-shaped object, and the arclength l is considered to be universal. The center angle θ is replacedby an inter-edge interval w, and assuming that the pair of edgesextracted by the change rate extraction unit 40 corresponds to both sidesurfaces of the truly rod-shaped object, the position extracted by thelower end region extraction unit 50 corresponds to the lower end of atruly rod-shaped object. In such a case, the change rate of thedistance, between the edge pairs and the change rate of the distancefrom the object to the fish-eye camera 80 is expected to be inverselyproportional.

Consequently, when the product of the change rate of the distancebetween the edge pairs and the change rate of the distance from theobject to the fish-eye camera 80 falls within a predetermined range, theobject detection unit 70, assuming that this pair of edges correspondsto the lower end of the rod-shaped object, determines that therod-shaped object exists. Conversely, when the product of the changerate of the distance between the edge pairs and the change rate of thedistance from the object to the fish-eye camera 80 does not fall withinthe predetermined range, if the position of the pair of edges or thelower end has not been found, the object detection unit 70 determinesthat the rod-shaped object does not exist.

[Apparatus Operation]

Next, operation of the object detection apparatus 100 according to thisexample embodiment will be described with reference to FIG. 9. FIG. 9 isa flowchart showing operation of the object detection apparatusaccording to an example embodiment of the invention. The followingdescription refers to FIG. 1 as appropriate. Also, in this exampleembodiment, an object detection method is implemented by operating theobject detection apparatus 100. Thus, the description of the objectdetection method in this example embodiment can be replaced with thedescription of operation of the object detection apparatus 100 below.

As shown in FIG. 9, first, the fish-eye image acquisition unit 10acquires a time series fish-eye image output from the fish-eye camera80, and inputs the acquired time series fish-eye image to the horizontalpanorama image generation unit 20 (step S1).

Next, the horizontal panorama image generation unit 20 converts the timeseries fish-eye image acquired in step S1 to a horizontal panoramaimage, for each frame included in that fish-eye image (step S2).

Specifically, in step S2, the viewpoint compensation vector generationunit 21 first generates a viewpoint compensation vector. Next, withrespect to the frame that was converted using the viewpoint compensationvector, the image generation unit 22 sets a plurality of viewpointsparallel to the ground plane in the horizontal direction of the frame.Then, the image generation unit 22, for each of the viewpoints that wasset, on the frame after conversion, based on a coordinate system thatincludes a sight line from each viewpoint as an axis, performsdistortion correction by perspective projection approximation.Furthermore, the image generation unit 22, using an image element in theperpendicular direction extracted from each of the frames aftercorrection, generates one new frame, and adopts this as the horizontalpanorama image.

Next, the edge pair extraction unit 30 extracts, for each frame, a pairof edges oriented in the perpendicular direction from the horizontalpanorama image (step S3).

Next, the change rate extraction unit 40 specifies, for each frame, theinter-edge distance between the pair of edges extracted in step S3, andextracts a change rate of the inter-edge distance between the frames(step S4).

Next, the lower end region extraction unit 50 extracts, for each frame,a region that corresponds to the lower end of the object predicted to beproviding the pair of edges, and specifies the position of the extractedregion in the horizontal panorama image (Step S5).

Next, the distance change rate extraction unit 60 calculates, for eachframe, the distance from the object to the fish-eye camera based on theposition specified in step S5, and furthermore extracts, between thesame frames as in step S4, the change rate of the distance from theobject to the fish-eye camera 80 (step S6).

Afterward, the object detection unit 70 determines whether or not theobject actually exists based on the change rate of the inter-edgedistance that was extracted by step S4 and the change rate of thedistance from the object to the fish-eye camera 80 that was extracted bystep S6 (step S7).

Advantageous Effects of Example Embodiment

As described above, according to this example embodiment, it is possibleto easily detect a rod-shaped object even in a case where a fish-eyecamera was used. The reason for this is that, in the horizontal panoramaimage, an object that extends in the vertical direction in the realspace is oriented in the perpendicular direction, and edges can beeasily extracted. Also, in the horizontal panorama image, the changerate of the inter-edge distance in the edge pair and the change rate ofthe distance from the object to the fish-eye camera can be easilyextracted, and based on the fact that there is a substantially inverserelationship between the two change rates, the existence or absence ofthe object can be determined.

Furthermore, in this example embodiment, by verifying the relationshipbetween the change rate of the inter-edge distance of the edge pair andthe change rate of the distance from the object to the fish-eye camera,it is possible to accurately determine whether or not the extracted edgepair was extracted from an actual rod-shaped object.

[Program]

A program according to this example embodiment may be a program thatcauses a computer to execute steps S1 to S7 shown in FIG. 9. Byinstalling this program in the computer and executing the program, theobject detection apparatus 100 and the object detection method accordingto this example embodiment can be realized. In this case, a processor ofthe computer performs processing to function as the fish-eye imageacquisition unit 10, the horizontal panorama image generation unit 20,the edge pair extraction unit 30, the change rate extraction unit 40,the lower end region extraction unit 50, the distance change rateextraction unit 60, and the object detection unit 70. Also, a computerinstalled in an automobile can be given as a specific example of thecomputer, but in this example embodiment, the computer is not limited tothat specific example. The computer may be a general-purpose computer,or, may be a computer installed in a home appliance, equipment forperforming work, or the like.

Also, the program according to this example embodiment may be executedby a computer system constructed using a plurality of computers. In thiscase, fur example, each computer may respectively function as any of thefish-eye image acquisition unit 10, the horizontal panorama imagegeneration unit 20, the edge pair extraction unit 30, the change rateextraction unit 40, the lower end region extraction unit 50, thedistance change rate extraction unit 60, and the object detection unit70.

Here, a computer that realizes the object detection apparatus 100 byexecuting the program according to this example embodiment will bedescribed with reference to FIG. 10. FIG. 10 is a block diagram showingan example of a computer that realizes the object detection apparatusaccording to an example embodiment of the invention.

As shown in FIG. 10, the computer 110 includes a CPU 111, a main memory112, a storage device 113, an input interface 114, a display controller115, a data reader/writer 116, and a communications interface 117. Theseunits are each connected so as to be capable of performing datacommunications with each other through a bus 121.

The CPU 111 opens the program (code) according to this exampleembodiment, which has been stored in the storage device 113, in the mainmemory 112 and performs various operations by executing the program in apredetermined order. The main memory 112 is typically a volatile storagedevice such as a DRAM (Dynamic Random Access Memory). Also, the programaccording to this example embodiment is provided in a state stored in acomputer-readable recording medium 120. Note that the program accordingto this example embodiment may be distributed on the Internet, which isconnected through the communications interface 117.

Also, other than a hard disk drive, a semiconductor storage device suchas a flash memory can be given as a specific example of the storagedevice 113. The input interface 114 mediates data transmission betweenthe CPU 111 and an input device 118, which may be a keyboard or mouse.The display controller 115 is connected to a display device 119, andcontrols display on the display device 119.

The data reader/writer 116 mediates data transmission between the CPU111 and the recording medium 120, and executes reading of a program fromthe recording medium 120 and writing of processing results in thecomputer 110 to the recording medium 120. The communications interface117 mediates data transmission between the CPU 111 and other computers.

Also, general-purpose semiconductor storage devices such as CF (CompactFlash (registered trademark)) and SD (Secure Digital), a magneticrecording medium such as a Flexible Disk, or an optical recording mediumsuch as a CD-ROM (Compact Disk Read-Only Memory) can be given asspecific examples of the recording medium 120.

Also, instead of a computer in which a program is installed, the objectdetection apparatus 100 according to this example embodiment can also berealized by using hardware corresponding to each unit. Furthermore, aportion of the object detection apparatus 100 may be realized by aprogram, and the remaining portion realized by hardware.

Some portion or all of the example embodiments described above can berealized according to (supplementary note 1) to (supplementary note 12)described below, but the below description does not limit the invention.

(Supplementary Note 1)

An object detection apparatus, including:

a fish-eye image acquisition unit configured to acquire a time seriesfish-eye image output from a fish-eye camera;

a horizontal panorama image generation unit configured to, for eachframe included in the time series fish-eye image, perform conversion toa horizontal panorama image in which a vertical direction in a realspace is expressed in a perpendicular direction of the frame, and anazimuth is expressed equiangularly in a horizontal direction of theframe;

an edge pair extraction unit configured to extract, for each frame, apair of edges in the perpendicular direction from the horizontalpanorama image;

a change rate extraction unit configured to extract, between the frames,a change rate of an inter-edge distance between the pair of edgesextracted;

a lower end region extraction unit configured to extract, for eachframe, a region that corresponds to a lower end of an object predictedto be providing the pair of edges;

a distance change rate extraction unit configured to calculate, for eachframe, a distance from the object to the fish-eye camera based on theposition of the region that corresponds to the lower end of the objectin the horizontal panorama image, and extract, between the frames, achange rate of the distance from the object to the fish-eye camera; and

an object detection unit configured to determine whether or not theobject exists based on the change rate of the inter-edge distance andthe change rate of the distance from the object to the fish-eye camerathat were extracted.

(Supplementary Note 2)

The object detection apparatus according to supplementary note 1,

wherein the horizontal panorama image generation unit,

based on a roll angle around an optical axis of the fish-eye camera anda pitch angle of the optical axis with respect to a plane parallel to aground plane of the object, generates a viewpoint compensation vectorthat converts the frame to an image obtained by capturing an image ofthe object from a direction parallel to the ground plane,

with respect to the frame that was converted using the acquiredviewpoint compensation vector, sets a plurality of viewpoints parallelto the ground plane in the horizontal direction of the frame,

for each of the viewpoints that was set, on the frame after conversion,based on a coordinate system that includes a sight line from theviewpoint as an axis, performs distortion correction by perspectiveprojection approximation, and

using an image element in the perpendicular direction extracted fromeach of the frames after correction, generates one new frame, and adoptsthis as the horizontal panorama image.

(Supplementary Note 3)

The object detection apparatus according to supplementary note 1 or 2,

wherein the object detection unit determines that the object exists whenthere is an inverse relationship between the change rate of theinter-edge distance and the change rate of the distance from the objectto the fish-eye camera.

(Supplementary Note 4)

The object detection apparatus according to any of supplementary notes 1to 3,

wherein the fish-eye camera is disposed such that, in a vehicle, theoptical axis direction is a direction inclined downward in a verticaldirection from a horizontal plane.

(Supplementary Note 5)

An object detection method, including:

(a) a step of acquiring a time series fish-eye image output from afish-eye camera;

(b) a step of, for each frame included in the time series fish-eyeimage, performing conversion to a horizontal panorama image in which avertical direction in a real space is expressed in a perpendiculardirection of the frame, and an azimuth is expressed equiangularly in ahorizontal direction of the frame;

(c) a step of extracting, for each frame, a pair of edges in theperpendicular direction from the horizontal panorama image;

(d) a step of extracting, between the frames, a change rate of aninter-edge distance between the pair of edges extracted;

(e) a step of extracting, for each frame, a region that corresponds to alower end of an object predicted to be providing the pair of edges;

(f) a step of calculating, for each frame, a distance from the object tothe fish-eye camera based on the position of the region that correspondsto the lower end of the object in the horizontal panorama image, andextracting, between the frames, a change rate of the distance from theobject to the fish-eye camera; and

(g) a step of determining whether or not the object exists based on thechange rate of the inter-edge distance and the change rate of thedistance from the object to the fish-eye camera that were extracted.

(Supplementary Note 6)

The object detection method according to supplementary note 5,

wherein in the (b) step,

based on a roll angle around an optical axis of the fish-eye camera anda pitch angle of the optical axis with respect to a plane parallel to aground plane of the object, a viewpoint compensation vector thatconverts the frame to an image obtained by capturing an image of theobject from a direction parallel to the ground plane is generated,

with respect to the frame that was converted using the acquiredviewpoint compensation vector, a plurality of viewpoints parallel to theground plane in the horizontal direction of the frame are set,

for each of the viewpoints that was set, on the frame after conversion,based on a coordinate system that includes a sight line from theviewpoint as an axis, distortion correction by perspective projectionapproximation is performed, and

using an image element in the perpendicular direction extracted fromeach of the frames after correction, one new frame is generated, andthis is adopted as the horizontal panorama image.

(Supplementary Note 7)

The object detection method according to supplementary note 5 or 6,

wherein in the (g) step, it is determined that the object exists whenthere is an inverse relationship between the change rate of theinter-edge distance and the change rate of the distance from the objectto the fish-eye camera.

(Supplementary Note 8)

The object detection method according to any of supplementary notes 5 to7,

wherein the fish-eye camera is disposed such that, in a vehicle, theoptical axis direction is a direction inclined downward in a verticaldirection from a horizontal plane.

(Supplementary Note 9)

A computer-readable recording medium that includes a program recordedthereon, the program including instructions that cause a computer tocarry out:

(a) a step of acquiring a time series fish-eye image output from afish-eye camera;

(b) a step of, for each frame included in the time series fish-eyeimage, performing conversion to a horizontal panorama image in which avertical direction in a real space is expressed in a perpendiculardirection of the frame, and an azimuth is expressed equiangularly in ahorizontal direction of the frame;

(c) a step of extracting, for each frame, a pair of edges in theperpendicular direction from the horizontal panorama image;

(d) a step of extracting, between the frames, a change rate of aninter-edge distance between the pair of edges extracted;

(e) a step of extracting, for each frame, a region that corresponds to alower end of an object predicted to be providing the pair of edges;

(f) a step of calculating, for each frame, a distance from the object tothe fish-eye camera based on the position of the region that correspondsto the lower end of the object in the horizontal panorama image, andextracting, between the frames, a change rate of the distance from theobject to the fish-eye camera; and

(g) a step of determining whether or not the object exists based on thechange rate of the inter-edge distance and the change rate of thedistance from the object to the fish-eye camera that were extracted.

(Supplementary Note 10)

The computer-readable recording medium according to supplementary note9,

wherein in the (b) step,

based on a roll angle around an optical axis of the fish-eye camera anda pitch angle of the optical axis with respect to a plane parallel to aground plane of the object, a viewpoint compensation vector thatconverts the frame to an image obtained by capturing an image of theobject from a direction parallel to the ground plane is generated,

with respect to the frame that was converted using the acquiredviewpoint compensation vector, a plurality of viewpoints parallel to theground plane in the horizontal direction of the frame are set,

for each of the viewpoints that was set, on the frame after conversion,based on a coordinate system that includes a sight line from theviewpoint as an axis, distortion correction by perspective projectionapproximation is performed, and

using an image element in the perpendicular direction extracted fromeach of the frames after correction, one new frame is generated, andthis is adopted as the horizontal panorama image.

(Supplementary Note 11)

The computer-readable recording medium according to supplementary note 9or 10,

wherein in the (g) step, it is determined that the object exists whenthere is an inverse relationship between the change rate of theinter-edge distance and the change rate of the distance from the objectto the fish-eye camera.

(Supplementary Note 12)

The computer-readable recording medium according to any of supplementarynotes 9 to 11,

wherein the fish-eye camera is disposed such that, in a vehicle, theoptical axis direction is a direction inclined downward in a verticaldirection from a horizontal plane.

Although the invention is described above with reference to exampleembodiments, the invention is not limited by the above exampleembodiments. Within the scope of the invention, various modificationsunderstandable by those skilled in the art can be made to theconfigurations or details of the invention.

INDUSTRIAL APPLICABILITY

As described above, according to the invention, it is possible to easilydetect a rod-shaped object from a captured image even in a case where afish-eye camera was used. The invention is useful in various fieldswhere a fish-eye camera is used.

REFERENCE SIGNS LIST

-   10 Fish-eye image acquisition unit-   20 Horizontal panorama image generation unit-   21 Viewpoint compensation vector generation unit-   22 Image generation unit-   30 Edge pair extraction unit-   40 Change rate extraction unit-   50 Lower end region extraction unit-   60 Distance change rate extraction unit-   70 Object detection unit-   100 Object detection apparatus-   110 Computer-   111 CPU-   112 Main memory-   113 Storage device-   114 Input interface-   115 Display controller-   116 Data reader/writer-   117 Communications interface-   118 Input device-   119 Display device-   120 Recording medium-   121 Bus-   201 Rod-shaped object-   301 Curved line segment in fish-eye image-   401 Straight line segment in horizontal panorama image-   601 Optical center (position where position of the fish-eye camera    is projected on the ground in the vertical direction)-   602, 605, 606 Virtual circles-   603 Azimuth at 9 o'clock when viewed from the optical center-   604 Azimuth at 3 o'clock when viewed from the optical center-   605, 606 Examples of lines equidistant from the optical center-   701, 702, 703 Horizontal lines

1. An object detection apparatus, comprising: a fish-eye imageacquisition unit configured to acquire a time series fish-eye imageoutput from a fish-eye camera; a horizontal panorama image generationunit configured to, for each frame included in the time series fish-eyeimage, perform conversion to a horizontal panorama image in which avertical direction in a real space is expressed in a perpendiculardirection of the frame, and an azimuth is expressed equiangularly in ahorizontal direction of the frame; an edge pair extraction unitconfigured to extract, for each frame, a pair of edges in theperpendicular direction from the horizontal panorama image; a changerate extraction unit configured to extract, between the frames, a changerate of an inter-edge distance between the pair of edges extracted; alower end region extraction unit configured to extract, for each frame,a region that corresponds to a lower end of an object predicted to beproviding the pair of edges; a distance change rate extraction unitconfigured to calculate, for each frame, a distance from the object tothe fish-eye camera based on the position of the region that correspondsto the lower end of the object in the horizontal panorama image, andextract, between the frames, a change rate of the distance from theobject to the fish-eye camera; and an object detection unit configuredto determine whether or not the object exists based on the change rateof the inter-edge distance and the change rate of the distance from theobject to the fish-eye camera that were extracted.
 2. The objectdetection apparatus according to claim 1, wherein the horizontalpanorama image generation unit, based on a roll angle around an opticalaxis of the fish-eye camera and a pitch angle of the optical axis withrespect to a plane parallel to a ground plane of the object, generates aviewpoint compensation vector that converts the frame to an imageobtained by capturing an image of the object from a direction parallelto the ground plane, with respect to the frame that was converted usingthe acquired viewpoint compensation vector, sets a plurality ofviewpoints parallel to the ground plane in the horizontal direction ofthe frame, for each of the viewpoints that was set, on the frame afterconversion, based on a coordinate system that includes a sight line fromthe viewpoint as an axis, performs distortion correction by perspectiveprojection approximation, and using an image element in theperpendicular direction extracted from each of the frames aftercorrection, generates one new frame, and adopts this as the horizontalpanorama image.
 3. The object detection apparatus according to claim 1,wherein the object detection unit determines that the object exists whenthere is an inverse relationship between the change rate of theinter-edge distance and the change rate of the distance from the objectto the fish-eye camera.
 4. The object detection apparatus according toclaim 1, wherein the fish-eye camera is disposed such that, in avehicle, the optical axis direction is a direction inclined downward ina vertical direction from a horizontal plane.
 5. An object detectionmethod, comprising: a acquiring a time series fish-eye image output froma fish-eye camera; a, for each frame included in the time seriesfish-eye image, performing conversion to a horizontal panorama image inwhich a vertical direction in a real space is expressed in aperpendicular direction of the frame, and an azimuth is expressedequiangularly in a horizontal direction of the frame; a of extracting,for each frame, a pair of edges in the perpendicular direction from thehorizontal panorama image; a extracting, between the frames, a changerate of an inter-edge distance between the pair of edges extracted; aextracting, for each frame, a region that corresponds to a lower end ofan object predicted to be providing the pair of edges; a calculating,for each frame, a distance from the object to the fish-eye camera basedon the position of the region that corresponds to the lower end of theobject in the horizontal panorama image, and extracting, between theframes, a change rate of the distance from the object to the fish-eyecamera; and a determining whether or not the object exists based on thechange rate of the inter-edge distance and the change rate of thedistance from the object to the fish-eye camera that were extracted. 6.The object detection method according to claim 5, wherein in theperforming conversion, based on a roll angle around an optical axis ofthe fish-eye camera and a pitch angle of the optical axis with respectto a plane parallel to a ground plane of the object, a viewpointcompensation vector that converts the frame to an image obtained bycapturing an image of the object from a direction parallel to the groundplane is generated, with respect to the frame that was converted usingthe acquired viewpoint compensation vector, a plurality of viewpointsparallel to the ground plane in the horizontal direction of the frameare set, for each of the viewpoints that was set, on the frame afterconversion, based on a coordinate system that includes a sight line fromthe viewpoint as an axis, distortion correction by perspectiveprojection approximation is performed, and using an image element in theperpendicular direction extracted from each of the frames aftercorrection, one new frame is generated, and this is adopted as thehorizontal panorama image.
 7. The object detection method according toclaim 5, wherein in the determining, it is determined that the objectexists when there is an inverse relationship between the change rate ofthe inter-edge distance and the change rate of the distance from theobject to the fish-eye camera.
 8. The object detection method accordingto claim 5, wherein the fish-eye camera is disposed such that, in avehicle, the optical axis direction is a direction inclined downward ina vertical direction from a horizontal plane.
 9. A non-transitorycomputer-readable recording medium that includes a program recordedthereon, the program including instructions that cause a computer tocarry out: a acquiring a time series fish-eye image output from afish-eye camera; a, for each frame included in the time series fish-eyeimage, performing conversion to a horizontal panorama image in which avertical direction in a real space is expressed in a perpendiculardirection of the frame, and an azimuth is expressed equiangularly in ahorizontal direction of the frame; a extracting, for each frame, a pairof edges in the perpendicular direction from the horizontal panoramaimage; a extracting, between the frames, a change rate of an inter-edgedistance between the pair of edges extracted; a extracting, for eachframe, a region that corresponds to a lower end of an object predictedto be providing the pair of edges; a calculating, for each frame, adistance from the object to the fish-eye camera based on the position ofthe region that corresponds to the lower end of the object in thehorizontal panorama image, and extracting, between the frames, a changerate of the distance from the object to the fish-eye camera; and adetermining whether or not the object exists based on the change rate ofthe inter-edge distance and the change rate of the distance from theobject to the fish-eye camera that were extracted.
 10. Thenon-transitory computer-readable recording medium according to claim 9,wherein in the performing conversion, based on a roll angle around anoptical axis of the fish-eye camera and a pitch angle of the opticalaxis with respect to a plane parallel to a ground plane of the object, aviewpoint compensation vector that converts the frame to an imageobtained by capturing an image of the object from a direction parallelto the ground plane is generated, with respect to the frame that wasconverted using the acquired viewpoint compensation vector, a pluralityof viewpoints parallel to the ground plane in the horizontal directionof the frame are set, for each of the viewpoints that was set, on theframe after conversion, based on a coordinate system that includes asight line from the viewpoint as an axis, distortion correction byperspective projection approximation is performed, and using an imageelement in the perpendicular direction extracted from each of the framesafter correction, one new frame is generated, and this is adopted as thehorizontal panorama image.
 11. The non-transitory computer-readablerecording medium according to claim 9, wherein in the determining, it isdetermined that the object exists when there is an inverse relationshipbetween the change rate of the inter-edge distance and the change rateof the distance from the object to the fish-eye camera.
 12. Thenon-transitory computer-readable recording medium according to claim 9,wherein the fish-eye camera is disposed such that, in a vehicle, theoptical axis direction is a direction inclined downward in a verticaldirection from a horizontal plane.